A Game Theory Approach for Estimating Reliability of Crowdsourced Relevance Assessments

In this article, we propose an approach to improve quality in crowdsourcing (CS) tasks using Task Completion Time (TCT) as a source of information about the reliability workers game-theoretical competitive scenario. Our is based on hypothesis that some are more risk-inclined and tend gamble with their use time when put compete other workers. This supported by our previous simulation study. We test 35 topics from experiments TREC-8 collection being assessed relevant or non-relevant crowdsourced both (referred “Game”) non-competitive “Base”) find competition changes distributions TCT, making them sensitive (i.e., wrong right) outcome non-relevant) assessments. also optimal function TCT weights weighted majority voting scheme. From probabilistic considerations, derive theoretical upper bound for performance cohorts 2, 3, 4, 5 workers, which criterion evaluate weighting achieves remarkable performance, significantly closing gap between accuracy obtained relevance judgements bound. Since takes advantage available quantity any CS tasks, believe it cost-effective and, therefore, can be applied assurance micro-tasks.